Evorus: A Crowd-powered Conversational Assistant Built to Automate Itself Over Time

Crowd-powered conversational assistants have been shown to be more robust than automated systems, but do so at the cost of higher response latency and monetary costs. A promising direction is to combine the two approaches for high quality, low latency, and low cost solutions. In this paper, we introduce Evorus, a crowd-powered conversational assistant built to automate itself over time by (i) allowing new chatbots to be easily integrated to automate more scenarios, (ii) reusing prior crowd answers, and (iii) learning to automatically approve response candidates. Our 5-month-long deployment with 80 participants and 281 conversations shows that Evorus can automate itself without compromising conversation quality. Crowd-AI architectures have long been proposed as a way to reduce cost and latency for crowd-powered systems; Evorus demonstrates how automation can be introduced successfully in a deployed system. Its architecture allows future researchers to make further innovation on the underlying automated components in the context of a deployed open domain dialog system.Comment: 10 pages. To appear in the Proceedings of the Conference on Human Factors in Computing Systems 2018 (CHI'18

GPT-4 as an Effective Zero-Shot Evaluator for Scientific Figure Captions

There is growing interest in systems that generate captions for scientific figures. However, assessing these systems output poses a significant challenge. Human evaluation requires academic expertise and is costly, while automatic evaluation depends on often low-quality author-written captions. This paper investigates using large language models (LLMs) as a cost-effective, reference-free method for evaluating figure captions. We first constructed SCICAP-EVAL, a human evaluation dataset that contains human judgments for 3,600 scientific figure captions, both original and machine-made, for 600 arXiv figures. We then prompted LLMs like GPT-4 and GPT-3 to score (1-6) each caption based on its potential to aid reader understanding, given relevant context such as figure-mentioning paragraphs. Results show that GPT-4, used as a zero-shot evaluator, outperformed all other models and even surpassed assessments made by Computer Science and Informatics undergraduates, achieving a Kendall correlation score of 0.401 with Ph.D. students rankingsComment: To Appear in EMNLP 2023 Finding

HeyTAP: Bridging the Gaps Between Users' Needs and Technology in IF-THEN Rules via Conversation

In the Internet of Things era, users are willing to personalize the joint behavior of their connected entities, i.e., smart devices and online service, by means of IF-THEN rules. Unfortunately, how to make such a personalization effective and appreciated is still largely unknown. On the one hand, contemporary platforms to compose IF-THEN rules adopt representation models that strongly depend on the exploited technologies, thus making end-user personalization a complex task. On the other hand, the usage of technology-independent rules envisioned by recent studies opens up new questions, and the identification of available connected entities able to execute abstract users' needs become crucial. To this end, we present HeyTAP, a conversational and semantic-powered trigger-action programming platform able to map abstract users' needs to executable IF-THEN rules. By interacting with a conversational agent, the user communicates her personalization intentions and preferences. User's inputs, along with contextual and semantic information related to the available connected entities, are then used to recommend a set of IF-THEN rules that satisfies the user's needs. An exploratory study on 8 end users preliminary confirms the effectiveness and the appreciation of the approach, and shows that HeyTAP can successfully guide users from their needs to specific rules

Imaging cell lineage with a synthetic digital recording system

Cell lineage plays a pivotal role in cell fate determination. Chow et al. demonstrate the use of an integrase-based synthetic barcode system called intMEMOIR, which uses the serine integrase Bxb1 to perform irreversible nucleotide edits. Inducible editing either deletes or inverts its target region, thus encoding information in three-state memory elements, or trits, and avoiding undesired recombination events. Using intMEMOIR combined with single-molecule fluorescence in situ hybridization, the authors were able to identify clonal structures as well as gene expression patterns in the fly brain, enabling both clonal analysis and expression profiling with intact spatial information. The ability to visualize cell lineage relationships directly within their native tissue context provides insights into development and disease

Understanding, Categorizing and Predicting Semantic Image-Text Relations

Two modalities are often used to convey information in a complementary and beneficial manner, e.g., in online news, videos, educational resources, or scientific publications. The automatic understanding of semantic correlations between text and associated images as well as their interplay has a great potential for enhanced multimodal web search and recommender systems. However, automatic understanding of multimodal information is still an unsolved research problem. Recent approaches such as image captioning focus on precisely describing visual content and translating it to text, but typically address neither semantic interpretations nor the specific role or purpose of an image-text constellation. In this paper, we go beyond previous work and investigate, inspired by research in visual communication, useful semantic image-text relations for multimodal information retrieval. We derive a categorization of eight semantic image-text classes (e.g., "illustration" or "anchorage") and show how they can systematically be characterized by a set of three metrics: cross-modal mutual information, semantic correlation, and the status relation of image and text. Furthermore, we present a deep learning system to predict these classes by utilizing multimodal embeddings. To obtain a sufficiently large amount of training data, we have automatically collected and augmented data from a variety of data sets and web resources, which enables future research on this topic. Experimental results on a demanding test set demonstrate the feasibility of the approach.Comment: 8 pages, 8 Figures, 5 table

Imaging cell lineage with a synthetic digital recording system

Multicellular development depends on the differentiation of cells into specific fates with precise spatial organization. Lineage history plays a pivotal role in cell fate decisions, but is inaccessible in most contexts. Engineering cells to actively record lineage information in a format readable in situ would provide a spatially resolved view of lineage in diverse developmental processes. Here, we introduce a serine integrase-based recording system that allows in situ readout, and demonstrate its ability to reconstruct lineage relationships in cultured stem cells and flies. The system, termed intMEMOIR, employs an array of independent three-state genetic memory elements that can recombine stochastically and irreversibly, allowing up to 59,049 distinct digital states. intMEMOIR accurately reconstructed lineage trees in stem cells and enabled simultaneous analysis of single cell clonal history, spatial position, and gene expression in Drosophila brain sections. These results establish a foundation for microscopy-readable clonal analysis and recording in diverse systems

Loss of Cofilin 1 Disturbs Actin Dynamics, Adhesion between Enveloping and Deep Cell Layers and Cell Movements during Gastrulation in Zebrafish

During gastrulation, cohesive migration drives associated cell layers to the completion of epiboly in zebrafish. The association of different layers relies on E-cadherin based cellular junctions, whose stability can be affected by actin turnover. Here, we examined the effect of malfunctioning actin turnover on the epibolic movement by knocking down an actin depolymerizing factor, cofilin 1, using antisense morpholino oligos (MO). Knockdown of cfl1 interfered with epibolic movement of deep cell layer (DEL) but not in the enveloping layer (EVL) and the defect could be specifically rescued by overexpression of cfl1. It appeared that the uncoordinated movements of DEL and EVL were regulated by the differential expression of cfl1 in the DEL, but not EVL as shown by in situ hybridization. The dissociation of DEL and EVL was further evident by the loss of adhesion between layers by using transmission electronic and confocal microscopy analyses. cfl1 morphants also exhibited abnormal convergent extension, cellular migration and actin filaments, but not involution of hypoblast. The cfl1 MO-induced cell migration defect was found to be cell-autonomous in cell transplantation assays. These results suggest that proper actin turnover mediated by Cfl1 is essential for adhesion between DEL and EVL and cell movements during gastrulation in zebrafish

Essential Domains of Anaplasma phagocytophilum Invasins Utilized to Infect Mammalian Host Cells

Anaplasma phagocytophilum causes granulocytic anaplasmosis, an emerging disease of humans and domestic animals. The obligate intracellular bacterium uses its invasins OmpA, Asp14, and AipA to infect myeloid and non-phagocytic cells. Identifying the domains of these proteins that mediate binding and entry, and determining the molecular basis of their interactions with host cell receptors would significantly advance understanding of A. phagocytophilum infection. Here, we identified the OmpA binding domain as residues 59 to 74. Polyclonal antibody generated against a peptide spanning OmpA residues 59 to 74 inhibited A. phagocytophilum infection of host cells and binding to its receptor, sialyl Lewis x (sLex-capped P-selectin glycoprotein ligand 1. Molecular docking analyses predicted that OmpA residues G61 and K64 interact with the two sLex sugars that are important for infection, α2,3-sialic acid and α1,3-fucose. Amino acid substitution analyses demonstrated that K64 was necessary, and G61 was contributory, for recombinant OmpA to bind to host cells and competitively inhibit A. phagocytophilum infection. Adherence of OmpA to RF/6A endothelial cells, which express little to no sLex but express the structurally similar glycan, 6-sulfo-sLex, required α2,3-sialic acid and α1,3-fucose and was antagonized by 6-sulfo-sLex antibody. Binding and uptake of OmpA-coated latex beads by myeloid cells was sensitive to sialidase, fucosidase, and sLex antibody. The Asp14 binding domain was also defined, as antibody specific for residues 113 to 124 inhibited infection. Because OmpA, Asp14, and AipA each contribute to the infection process, it was rationalized that the most effective blocking approach would target all three. An antibody cocktail targeting the OmpA, Asp14, and AipA binding domains neutralized A. phagocytophilumbinding and infection of host cells. This study dissects OmpA-receptor interactions and demonstrates the effectiveness of binding domain-specific antibodies for blocking A. phagocytophilum infection